Optimization by Variational Bounding

We discuss a general technique that forms a differentiable bound on non-differentiable objective functions by bounding the function optimum by its expectation with respect to a parametric variational distribution. We describe sufficient conditions for the bound to be convex with respect to the variational parameters. As example applications we consider variants of sparse linear regression and SVM training. 1 Variational Optimization We consider the general problem of function minimization, min x f(x) for vector x. When f is differentiable and x continuous, optimization methods that use gradient information are typically preferred over non-gradient based approaches since they may take advantage of a locally optimal direction in which to search. However, in the case that f is not differentiable or x is discrete, gradient based approaches are not directly applicable. In that case, alternatives such as relaxation, coordinatewise optimization and stochastic approaches are popular. Our interest is to discuss another general class of methods that yield differentiable surrogate objectives for discrete x or non-differentiable f . The Variational Optimization (VO) approach is based on the bound f∗ = min x∈C f(x) ≤ 〈f(x)〉 p(x|θ) ≡ E(θ) where 〈·〉p denotes the expectation with respect to the probability density function p defined over the solution space C. The parameters θ of the distribution p(x|θ) can then be adjusted to minimize the upper bound E(θ). This bound can be trivially made tight provided the distribution p(x|θ) is flexible enough to allow all its mass to be placed in the optimal state x∗ = argminx∈C f(x). The variational bound is equivalent to an objective smoothed by convolution with the variational distribution, with the degree of smoothing increasing as the dispersion of the variational distribution increases. The gradient of E(θ) is given by